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Abstract 


The rapid advances in information and communication technologies, coupled with 
increased access to information and the formation of global communities, have resulted 
in interest among researchers and academics to revise educational practice to move 
beyond traditional ‘literacy’ skills towards an enhanced set of “multiliteracies” or “new 
media literacies”. Measuring the literacy of a population, in the light of its linkage to 
individual and community wealth and wellbeing, is essential to determining the impact 
of compulsory education. The opportunity now is to develop tools to assess individual 
and societal attainment of these new literacies. Drawing on the work of Jenkins and 
colleagues (2006) and notions of a participatory culture, this paper proposes a 
conceptual framework for how learning analytics can assist in measuring individual 
achievement of multiliteracies and how this evaluative process can be scaled to provide 
an institutional perspective of the educational progress in fostering these fundamental 
skills. 
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Introduction 


The development of a literate population is one of the most recognizable goals of public 
education. Central to this goal has been defining what counts as literacy for 
contemporary society. In the more traditional sense, literacy was, and continues to be, 
deeply enmeshed with the written word (Kress, 2003). As Jim Dator (2005) argues, 
“seldom has a technology been the subject of more worship than the word is in literate 
cultures” (p. 202). This emphasis on the written word stems from the perceived 
relationship between basic literacy skills and future economic prospects. Even in the 
industrial age, the cultivation of a literate population (reading, writing and basic 
arithmetic) was viewed as an essential ingredient for achieving democracy, economic 
growth, and social stability (Kalman, 2008). In essence, an individual’s future social and 
economic prosperity was related to their capacity to read and write (Leadbeater & 
Wong, 2009). It is little wonder then that literacies have continued to be so intimately 
linked to economic benefits, such as higher socio-economic status, increased job 
opportunities, and increased wealth within the community (UNESCO Education Sector, 
2004). 

Basic literacies are as relevant today in the information age as they were previously, 
especially given that the Internet and mobile devices remain as heavily text-based 
technologies (Greenhow & Robelia, 2009; Warschauer, 2007). However, the 
pervasiveness and pace of change associated with new forms of media and the contexts 
in which they are applied places additional expectations on what it now means to be 
“literate” (Anstey & Bull, 2006; Huijser, 2006). That is, the expectation now exists that 
individuals have at their disposal a diverse set of skills and cultural competencies 
necessary to navigate the various forms of digital communication and participation in a 
global society. While not understating the continued importance for reading, writing, 
and numeracy skills, an emphasis placed solely on those ‘basics’ ignores the broader 
changes that have occurred in the cultural, social, and economic landscape and the 
speed of access to technologies and information that are now prevalent in today’s 
society (Kalantzis, Cope, & Harvey, 2003). A society that is increasingly reliant on 
technology for information access and communication alongside the globalization of 
information requires its citizens to effectively utilize a greatly expanded skill set to 
encompass cultural and new media competencies as well as a recognition of the various 
contexts in which they are applied (Coiro, Knobel, Lankshear, & Leu, 2014). 

If an expanded form of literacy, multiliteracies, is the key to individual and community 
wealth and wellbeing within a society, then it is crucial to the value of education to 
establish measures regarding how well students in the education system are mastering 
these fundamentals. Not only is there a need to assess individual students on their 
progress and mastery of skills in multiliteracy, but it is important to measure systemic 
progress and attainment of multiliteracies of the education system and society as a 
whole. Measurement of the literacy of a population, in the light of its linkage to 
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individual and community wealth and wellbeing, is an essential and fundamental 
measure of the success of formal education. 

This paper begins with an examination of multiliteracies in order to create a defined set 
of skills that can be considered essential literacies for today’s society. Following this 
review, the potential assessment and learning activity artifacts that might be generated 
in the process of learning multiliteracies are discussed. The trend for increasing 
adoption of online and blended modes of learning continues unabated at a global level 
and as such there are now unprecedented levels of trace data that can be harnessed to 
inform teaching and learning practice. Given the potential artifacts that learners 
generate in these online learning environments, this paper proposes a conceptual 
framework for using learning analytics to measure the development of multiliteracies 
across a group of learners. This paper outlines a framework for moving analytics from 
what has been previously described as the “low hanging fruit” 1 up the ladder to richer 
and more complex multi-dimensional analytics in education. The conceptual framework 
proposed in this paper emphasizes the identification of key multiliteracies based on a 
review of frameworks created by media scholars and then suggests possible data sources 
and analytics strategies that will enable a means of measuring and determining the 
impact of those literacies. Validation of the learning analytics/multiliteracy framework 
is beyond the scope of this paper but it is expected that detailing the relationship 
between literacies, artefacts, and analytics techniques will lead to empirical work to test, 
validate, and revise the frameworks offered by media scholars. 


Multiliteracies 


Recognition of the dramatically changing nature of what it means to be literate in the 
so-called ‘information age’ has seen the rise of discussions within educational research 
around the importance of students developing “multiliteracy” skills (I. Brown, Lockyer, 
& Caputi, 2010; Cope & Kalantzis, 2000a; Haythornthwaite & Andrews, 2011). As first 
described by the New London Group (1996), the term multiliteracies extends the scope 
of traditional literacy to include the diversity of media and modes of communication 
that are now available to learners and the varying contexts in which they are utilized. In 
discussing the genesis of the term multiliteracies, Gee (2009) noted that literacy 
“needed to be viewed as embedded in multiple socially and culturally constructed 
practices, not seen as a uniform set of mental abilities or processes” (p. 196). In essence, 
the New London Group (NLG) challenged the existing singular view of the term literacy 
recognizing the multiplicity of communications available to learners both as producers 
and consumers and the increasing cultural and linguistic diversity that is prevalent 
today (Cope & Kalantzis, 2000b). Simply put, new technologies are changing the way we 


1 Learning and Knowledge Analytics (2011): Knewton - the future of education? 
http://www.learninganalvtics.net/?p=126 
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communicate and interact. As such, the term multiliteracies is often used 
interchangeably with new literacies, digital literacies, or media literacies. 

Since establishing “a pedagogy of multiliteracies” (Cope & Kalantzis, 2000b; New 
London Group, 1996), much conceptual and theoretical work has been undertaken in 
the literacy field (e.g., Gee, 2007; Kress, 2003; Lankshear & Knobel, 2003; Muspratt, 
Luke, & Freebody, 1998). The goal of this paper is to bring a new perspective of how to 
assess and evaluate the development of new literacies and the pedagogical activities that 
underpin such characteristics and skills, rather than to establish an authoritative 
definition or review of literacies. 

A Participatory Culture 

While further theorizing of the changing nature of and intersection between literacy, 
learning, and digitality needs to continue, there is acceptance that multiliteracies 
involve an increasing set of social skills that draw upon an ever expanding set of 
technologies, media, and discourses (Gee, 2007; Unsworth, 2001). In this context, 
viewing literacy through a sociocultural lens encompasses the fundamentals of literacy 
(e.g., reading, writing and meaning making) within embedded social practice 
(Lankshear & Knobel, 2007). Henry Jenkins (2006) emphasizes this point in noting that 
“the new literacies almost all involve social skills developed through collaboration and 
networking” (p. 4). 

The complexity of the information, media, and technology environment that learners 
draw on in their day to day academic and social activities, as well as the types of skills 
necessary to be productive community members, is captured in Jenkins, et al.’s (2006) 
notion of a participatory culture. Jenkins and colleagues describe a participatory culture 
as one with “relatively low barriers to artistic expression and civic engagement, strong 
support for creating and sharing creations, and some type of informal mentorship 
whereby experienced participants pass along knowledge to novices” (p. xi). Educational 
engagement in this form of practice is strongly aligned with more Vygotskian-influenced 
approaches to learning such as social learning (J. S. Brown & Adler, 2008), game-based 
learning (Gee, 2007), self-directed learning (Garrison, 1997), and communities of 
practice (Lave & Wenger, 1991). The US Department of Education report, Transforming 
American Education: Learning Powered by Technology (2010), echoes this sentiment 
in noting that contemporary education practice must embrace technology mediated 
modes of learning for establishing access to diverse resources and connections to the 
broader learning community - a community that extends beyond the classroom and 
beyond national borders. Advocacy for these learning orientations is grounded in the 
view that knowledge is constructed through an individual’s interactions with the 
broader social group. To be productive participants in these social learning approaches 
requires not only the basic literacies but what Jenkins describes as “new media 
literacies” (Jenkins et al., 2006) or “cultural competencies and social skills” (p. xiii) 
necessary to be productive participatory citizens. 
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The competencies and skills necessary for a participatory culture include networking, 
collaboration, creativity, citizenship, and communication within a multiplicity of modes 
and mediums. Table 1 outlines Jenkins et al.’s (2006) new media literacies stressing the 
transition of literacy practice from individual to community. Simply put, for 21 st century 
education these skills and attributes are considered to be the ‘new basics’. As such, the 
plurality of these new literacies also calls for new and diverse forms of assessment to be 
developed and implemented within the formal education context. 

Table 1 

Overview of New Media Literacies (Jenkins et al., 2006) 


New media literacies 

Description 

Play 

Capacity to experiment with one’s surroundings as a form of 
problem-solving. 

Performance 

Ability to adopt alternative identities for improvisation and 
discovery. 

Simulation 

Ability to interpret and construct dynamic models of real- 
world processes. 

Appropriation 

Ability to meaningfully sample and remix media content 

Multitasking 

Ability to scan one’s environment and shift focus as needed to 
salient details. 

Distributed cognition 

Ability to interact meaningfully with tools that expand mental 
capacities. 

Collective intelligence 

Ability to pool knowledge & compare notes with others toward 
a common goal. 

Judgment 

Ability to evaluate the reliability & credibility of different 
information sources. 

Transmedia 

Ability to follow the flow of stories and information across 

Navigation 

multiple modalities. 

Networking 

Ability to search for, synthesize, and disseminate information. 

Negotiation 

Ability to travel across diverse communities, discerning and 
respecting multiple perspectives, and grasping and following 
alternative norms. 


Assessment of Literacies 


The availability of an ever expanding wealth of information online, alongside the growth 
in the adoption of web enabled mobile platforms, impacts on how, when, and where we 
learn, who we learn with, and how we evaluate the legitimacy of information 
(Haythornthwaite & Andrews, 2011). Anywhere anytime learning now occurs with 
potential learners engaging with mentors, peers, and experts with minimal regard to 
time or place. These learner empowered collaborations complement recent research 
related to teaching and learning where social learning is viewed as the primary 
framework for effective pedagogical practice (J. S. Brown & Adler, 2008; Siemens, 
2005). While embracing the pedagogical benefits that technology brings to bear to the 
education context, there is an associated necessity for developing “new and better ways 
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to measure” learning (U.S. Department of Education Office of Educational Technology, 
2010, p.xi). 

Government calls for education accountability through standardized testing has in part 
reinforced the concept of a ‘back to basics’ literacy approach. Standardized testing using 
more traditional assessment methods drastically limits the evaluation of the social and 
cultural skills and competencies associated with participatory skills. This point is 
emphasized by Mary Kalantzis and colleagues (2003) in noting that “accountability and 
commensurability has focused global attention on producing education outcomes which 
are simple to interpret, tangible and transparent, and easily comparable” (p. 15). There 
is a need to examine alternate assessment approaches that can complement calls for 
accountability and standardization against an imperative for implementing new 
approaches that can measure the full spectrum of multiliteracies. More recently, Literat 
(2014) noted that the few studies that have attempted to measure literacy have generally 
focused on an individual’s understanding of text and or audio. Literat argued that 
alternate assessment methods are required to evaluate a broader suite of literacies. 
Literat addressed this methodological problem by demonstrating the potential for more 
psychometric methods to provide potential indicators of new media literacy skills. 
Literat’s (2014) new media literacy questionnaire begins to illustrate the capacity for 
alternate approaches to identity potential lead indicators of literacy development and 
comprehension. As detailed in Table 1 many of the skills are not readily assessed 
through standardized testing. Literacies such as “play” or “negotiation” are socially 
embedded and enabled. As a result, effective assessment requires a more nuanced and 
diverse approach than is possible through basic knowledge testing. 

Alongside the curricular adoption of the skills and competencies associated with 
multiliteracies, current assessment practices must extend beyond the measurement of 
outputs to provide greater insight into learning processes (Edwards, 1997). To facilitate 
this transition, David Boud (2000) maintained that assessment practices cannot remain 
the “exclusive domain of assessors” (p. 151) as responsibility also resides with the 
learner. It is only through an individual’s capacity to evaluate, interpret, and make 
decisions about their learning progress will education foster the skills necessary for 
capable lifelong learners (Boud, 2000). Boud’s notion of “sustainable assessment” and 
learner empowerment is in keeping with calls for the development of new assessment 
practices that address literacies such as creativity, networking, citizenship, and 
collaboration (Kalantzis et al., 2003). Thus, for education, contemporary assessment 
practice needs to reflect these community-centric learning models to include measures 
of learning progression at both the community and individual levels. This is exemplified 
in Csikszentmihalyi’s (2006) call for creativity to be assessed at the level of community 
not as an end product derived from a sole individual. However, pressures of 
accountability, equity, standards, and quality assurance results in a loss of focus on 
assessment of community in lieu of an individual. It is at this intersection of education 
accountability and need for collaborative real-time assessment measures that analytics 
can provide an important role. The following section provides a brief overview of 
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learning analytics before discussing the alignment of Jenkin’s (2006) new media 
literacies with learning analytics indicators. This example is used to illustrate how 
analytics can provide insight into individual and community based learning for the 21 st 
century. 


What are Learning Analytics? 


The field of education has entered the era of “big data”. The Mckinsey report (2011) 
defines “big data” as a “dataset whose size is beyond the ability of typical database 
software tools to capture, store, manage and analyze” (p. 1). As noted in the report, 
while the definition is subjective, the ‘big data’ concept relates to the flood of data that is 
generated and captured as users interact with the myriad of IT systems that support 
daily activities from iTunes, Twitter, and YouTube through to ecommerce and public 
services. In an education context, student information systems and learner interactions 
with various technologies such as learning management systems (LMS) and social 
media leave a trail of digital breadcrumbs that are accessible for data mining and 
analysis (Buckingham Shum & Ferguson, 2012; Duval, 2011; Fournier, Kop, & Sitlia, 
2011; Macfadyen & Dawson, 2010). The extraction and analysis of the data derived from 
these technologies has captured the attention of politicians, education leaders, 
researchers, and day to day education practitioners. This is partly due to the expectation 
that these relatively new forms of, and processes for, analytics can address some of the 
more pressing concerns confronting the education sector. These concerns include 
increasing completion rates, addressing basic curriculum standards, student 
questionnaire overload, accountability and measuring teaching effectiveness and 
quality. Although to date the field of learning analytics has largely focused on learner 
progression and developing lead indicators of student attrition (Campbell, De Blois, & 
Oblinger, 2007; Fritz, 2011), there is much potential (and some early research) in using 
student interaction data to establish indicators of more complex concepts such as 
knowledge construction (Pozzi, Manca, Persico, & Sarti, 2007), sense of community 
(Dawson, 2006; Gasevic, Adesope, Joksimovic, & Kovanovic, submitted), creativity 
(Dawson, 2010; Dawson, McWilliam, & Tan, 2011), and self-regulated learning (Biswas, 
Jeong, Kinnebrew, Sulcer, & Roscoe, 2010; Winne & Hadwin, 2013). 

Learning analytics adopts many of the methods and approaches used by “big data” 
practitioners, but is specifically defined as “the measurement, collection, analysis and 
reporting of data about learners and their contexts, for the purposes of understanding 
and optimizing learning and the environments in which it occurs” (Siemens & Long, 
2011). Learning analytics uses the data associated with a learner’s interactions with 
content, other learners, and the educational institution to make decisions and 
evaluations about teaching practices, personalized content, and needed interventions 
for learner success. The field draws on and integrates research and methodology related 
to data mining, social network analysis, data visualization, machine learning, learning 
sciences, psychology, semantics, artificial intelligence, e-learning, and educational 


|Vol 15 | N04 


Creative Commons Attribution 4.0 International License 


Sept./i4 


290 






Analytics to Literacies: The Development of a Learning Analytics Framework for Multiliteracies Assessment 


Dawson and Siemens 


theory and practice. Learning analytics focuses on the interpretation of the educational 
data from a learner and teacher orientation. This places as much emphasis on 
understanding the pedagogical context from where the data is derived as developing 
statistically robust interpretive and predictive models. 

The interest in learning analytics is fuelled by increasing calls for accountability, quest 
for evidence, and the demonstration of what counts as learning and teaching quality 
alongside decreasing fiscal resources. Similarly, the accessibility to student learning 
data from various information systems has also contributed to the use of learning 
analytics as a tool for measuring impact and informing the strategic decision making 
process within education organizations (Macfadyen & Dawson, 2012). Although the 
concept of using these forms of analytics for informing education practice has been well 
received, the majority of educational institutions seldom make optimal use of their 
available data and analytical resources (Norris, Baer, Leonard, Pugliese, & Lefrere, 
2008). Furthermore, as a result of the relative infancy of the field of learning analytics, 
the level and sophistication of the data analysis performed has to date been limited, with 
a predominance of studies and reports undertaking simple univariate or bivariate 
analyses (Dawson, Gasevic, Siemens, & Joksimovic, 2014). For instance, low-level 
analysis such as the reporting of student login times, number of posted messages in a 
discussion forum, or total time online are common analytics approaches. Bivariate 
analyses have tended to relate the readily accessible learning management system 
(LMS) data (e.g., number of messages posted, frequency of logins, time online, etc.) to 
academic performance as measured by grades. 

These early studies have served well to raise the profile of learning analytics and to 
generate interest in the field as a means to improve student retention and success, 
including the development of early indicators of academic performance through LMS 
usage trends. The Purdue University course Signals project 2 is an example of early 
warning systems. The Signals software incorporates student past academic 
performance, demographics with LMS activity, and engagement to provide a statistical 
assessment of an individual learner’s probability for success (Arnold, 2010). Watson 
and Gemin (2008) defined "at-risk" as students with a high probability of withdrawing 
from the course as a result of academic failure, dropping out, or expulsion for behavioral 
reasons. In the instance of Signals, success is measured by an individual’s overall 
assessment score. The intent of the software is to provide early warning indicators to 
learners and instructors regarding an individual’s risk of course failure or attrition. 

The implementation of practices that focus on improving student retention has a direct 
and easily measurable return on investment and aligns well with numerous federal 
government initiatives internationally. For instance, the Australian Bradley Report 
(Bradley, Noonan, Nugent, & Scales, 2008) emphasized the imperative for increasing 
graduation rates; Singapore’s “Thinking Schools, Learning Nation” (Ministry of 
Education, 1998) called for evidence based policy making; and the US “Building a Grad 


2 Purdue University Course Signals: http://www.itap.purdue.edu/learnine/tools/signals/ 
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Nation” (Balfanz, Bridgeland, Moore, & Fox, 2010) outlined a strategy for addressing 
the nation’s increasing level of attrition. These types of reports add further weight to the 
importance of establishing well-grounded processes and practices for informed decision 
making practice and quality assurance and accountability. However, stated targets for 
reducing attrition alone, while admirable, takes a simplistic view of the learning process 
and does little to measure learning of skills and attributes. 

Researchers such as Macfadyen and Dawson (2010) have investigated LMS data to 
predict student academic performance. While these authors used relatively simple 
metrics such as student grades as an indicator of success, the study incorporated a more 
sophisticated modeling process to analyze 15 LMS variables to determine and evaluate a 
best-fit predictive model. The authors note that social learning (such as in a discussion 
forum) and formative assessment tasks completed accounted for greater than 30% of 
the variation in student grade. Macfadyen and Dawson’s study moved beyond the 
incorporation of simple engagement measures such as number of discussion messages 
posted towards more complex analytic measures such as social network analysis (SNA). 
The inclusion of social network methods for determining community participation 
aligns with the move for new media literacies to be evaluated at the level of community 
(Jackson, 2006). Combining SNA with other more automated qualitative analytics 
through machine learning and computational linguistics can further enrich the 
prospects for establishing a meaningful model for evaluating new literacies. For 
example, while SNA provides an indication of the strength and diversity of relationships 
an individual actor establishes in a network there is minimal reference to be able to 
identify the purpose and value of this relationship. In this context, the inclusion of 
automated content analysis provides added insight that can determine the extent to 
which an individual demonstrates good participatory practice. More recently, learning 
analytics research has started to transition to more sophisticated methods targeting 
discourse, language, and affective learner attributes'?. 


Literacies and Learning Analytics 


The inclusion of methods such as SNA, epistemic network analysis (Shaffer et al., 
2009), and affective learner attributes (Baker, D'Mello, Rodrigo, & Graesser, 2010) 
merged with computational linguistics moves the focus of learning analytics from the 
measurement of an end product towards an evaluation of the process of learning. It is 
unlikely such a model can be developed through a reliance on extracting student 
assessment data and LMS activity alone. Grades and LMS activity do not sufficiently 
represent the diversity of social and cultural based interactions students frequently 
engage in as learning is not the sole domain of formal institutions. This calls for an 
examination of how student engagement across multiple educational and social systems 
can be captured and incorporated, including the more qualitative artifacts such as 


3 http://machineanalvtics.org/schedule/ 
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student discussion postings, essay writing, blog posts, YouTube, Facebook or Twitter 
feeds. The inclusion of social network methodologies provides a rich stream of data and 
an important analytical view to determine the types of relationships and extent of 
participation in defined communities. In this context, analytics can begin to capture the 
necessary insights that relate the individual to the community - or in the terms of 
Jenkins, the individual’s active engagement in a participatory culture. The following 
section outlines possible metrics for evaluating multiliteracies. This is not to suggest 
that existing forms of student assessment should be replaced, but rather the section 
outlines additional indicators that can further complement the suite of assessment 
practices regarding an individual’s progression and therefore the broader 
demonstration of new media literacies. 


Using Analytics to Assess Multiliteracies 


As with any kind of evaluative measurement, defining an outcome or process’s success 
indicators will guide the selection of tools for assessment. For the purposes of this 
paper, Jenkins et al.’s proposed new media literacies and definitions (Table 1) will be 
used as the basis for the learning analytics framework. Jenkins et al.’s multiliteracies 
have been clustered in order to refine the types of analytical data that can provide 
sufficient lead indicators of competency. For example, play, performance, and 
simulation are closely linked in terms of their affinity with problem solving processes, 
experimentation, and risk taking. This may be realized through activities such as gaming 
or role play. However, simulation also corresponds to aspects of appropriation, whereby 
these skills involve a form of creation or co-creation. Items such as collective 
intelligence, judgment, and negotiation relate to accessing, sharing, and evaluating 
information and resources within and across networks. The skill of distributed cognition 
lies in the intersection between accessing and sharing information and navigation and 
multi-tasking. Figure 1 presents a visualization of the associated grouping of Jenkins et 
al.’s (2006) classification of media literacies. The four clusters described above can be 
measured, monitored, and reported through a diversity of analytics and modified for the 
specific pedagogical context. 
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Figure 1. Clustering of media literacies. Adapted from Jenkins et al. (2006). 


Experimentation: Play, Performance, and Simulation 

Experimentation can be determined through the diversity of user interactions in 
particular technologies. This may include online games, role playing, or use of assigned 
simulations. There is a trend for educators to adopt “serious games” and simulations to 
support the achievement of stated learning objectives (Moreno-Ger, Burgos, Martlnez- 
Ortiz, Sierra, & Fernandez-Manjon, 2008). In this context, the use of virtual worlds 
provides an opportunity for learner experimentation in an immersive environment (De 
Freitas, Rebolledo-Mendez, Liarokapis, Magoulas, & Poulovassilis, 2010). User 
interactions mined from these forms of game based engagement can provide insight 
into the learner’s competency for play and performance. For example, the degree to 
which the user engages with the resources, models, pathways, and specific goal oriented 
outcomes provides an indication of the user’s efficacy for engaging in play activity as 
well as adopting alternate persona for the purposes of discovery, reflection, and 
perspective. 

Products/Creation: Appropriation and Simulation 

Products/creation can be observed directly through the generation of specific artifacts 
such as multimediae As such these products, associated methods for creation, and level 

4 See for example http://dsl06.us/ as an artifact-creating online community 
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of co-creation and engagement can be measured. In Axel Bruns’ (2008) terms, the 
concept of prod-users defines a shift away from a model of production to a more 
collaborative and user-led model of creation. The concept reinforces the notion that any 
digital product can be remixed and repurposed, and as such is a continual state of flux 
and evolution. The feedback link from producer to consumer back to producer can be 
collaborative and completed within an extremely short time frame. Wikipedia illustrates 
this dynamic and evolving collaborative system. Essentially, the flow and utilization of 
products as they evolve in both social and cultural importance can act as an indicator of 
appropriation and simulation. 

Network Agility and Citizenship: Collective Intelligence, 
Judgment, Negotiation, Distributed Cognition 

The skills grouped into network agility/citizenship represent an individual’s level of 
competence and capacity for building relationships, participating in networks, and 
contributing to a community of learners. This cluster relates to a learner’s role, position, 
and contributions to the learning network. This can be readily measured via social 
network analysis (SNA). The integration of SNA not only provides insight into the 
strength and diversity of relationships formed but also the types of information or 
resources shared within the social system (Haythornthwaite, 2002). 

In examining the impact of network position and the flow of good ideas Ronald Burt 
(2004) illustrated the value of SNA to provide insights into network position and 
information and resource access. In this example, Burt noted that actors bridging two or 
more previously disparate network clusters demonstrate greater agility and enterprise 
than peers positioned within an insular network cluster. According to Burt (2004), these 
individuals “are able to see early, see more broadly, and translate information across 
groups” (p. 354). Burt sees this ‘translating’ function as value-adding creativity. This is 
not just because of the extent to which ‘brokers’ are able to move knowledge around in 
value-adding ways, but also their capacity to build, sustain, and expand upon their 
networks within and outside the existing environment. The skills these “border 
crossers” (McWilliam & Dawson, 2008) exhibit in order to establish these diverse 
networks reflect a high level of competence with digital literacies and also demonstrate 
good participatory practice. Hence understanding can be gained about how social 
network analysis can assess and provide early indicators of a participatory culture. 

Interactions in networks are often more complex than the exchange of information 
between two or more individuals. Multidimensional networks (Contractor, Monge, & 
Leonardi, 2011) reflect the more complex interactions that are formed between different 
technologies and people. As the use of social technologies continues to penetrate the 
education sector there is an associated increase in opportunity to extract and visualize 
the relationship data. Aligning SNA or multidimensional network activity with new 
media literacies, an individual’s position in the network and the diversity of 
relationships formed serves as a measure of their capacity to form relationships, provide 
and share resources and information (collective intelligence, distributed cognition) as 
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well as negotiate, adapt, and respect social and cultural community norms (judgment, 
negotiation). However, while SNA provides a robust approach for ascertaining an 
individual’s network agility there is limited information regarding the quality of the 
relationships and resources accessed and shared. It is only through an examination of 
the quality of the exchanges can we begin to evaluate the proficiency of an individual’s 
participatory practices. 

The use of content analysis alongside SNA affords rich insights into the quantitative as 
well as qualitative aspects of a social system. Content analysis is a commonly adopted 
approach for determining the perceived quality of the knowledge construction process 
occurring within a learning network (De Laat, 2002; De Laat, Lally, Lipponen, & 
Simons, 2006). However, the mapping of the captured exchanges to a pre-defined 
coding scheme has to date been largely a manual and time consuming process. Hence, 
the level of integration between SNA and content analysis methods has thus far been 
minimal at best. The adoption of machine learning techniques and tools such as 
TagHelper (Rose et al., 2007) and Cohere (De Liddo, Buckingham Shum, Quinto, 
Bachler, & Cannavacciuolo, 2011) demonstrate the potential for automating the coding 
process and therefore the potential for measuring an individual’s network agility and 
citizenship. 

Task Effectiveness and Efficiency: Multi-Tasking, Networking, 
Trans-Media Navigation 

Measures of task effectiveness and efficiency relate specifically to the choice and 
method for achieving goals and outcomes as well as overall comprehension. For 
instance, transmedia navigation requires a level of proficiency for interpreting and 
understanding different representations of information or social and cultural icons and 
artifacts across and within multiple domains. Networking refers to an individual’s 
competency with various tools and methods for searching, synthesizing, and 
disseminating information. These characteristics imply a high level of competence and 
efficiency with accessing tools and resources. As such, the tool selection, search data and 
techniques, and levels of engagement reflect analytics data to evaluate the 
characteristics underpinning this cluster. Kennedy and Judd (2011; 2004) examined 
digital audit trails of a student’s activity within various technologies in order to identify 
specific patterns of behavior. The authors concluded that the audit trails provided 
significant interpretive power regarding an individual’s learning behavior, search 
process, and tool selection. By further incorporating a level of semantic analysis it is 
possible to also establish a user’s search term technique based on specific request and 
tasks. These forms of analysis provide a measure of a user’s competency with 
information evaluation and use in different settings and across various media. 
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Table 2 

Artifacts Generated for Assessment as Exhibited by Different Literacies 


New media 
literacies 


Play 


Performance 

Simulation 

Appropriation 

Multitasking 


Distributed 

cognition 

Collective 

intelligence 

Judgment 

Transmedia 

navigation 

Networking 

Negotiation 


Examples of artifacts produced 


Interactions generated in game or simulation settings, evidence of 
experimentation through novel combinations of game artifacts 
(Shaffer et al., 2009) 

Video, creations, digital artifacts, activity in case study 
Recordings of activity within simulations, ability to review, reflect, 
and peer critique 

Using, combining, and remixing digital artifacts from other 
learners 

Data collected from learners activity in a virtual machine 
environment (Pardo & Kloos, 2011), recordings made with tracking 
software 

Use of range of digital tools for information search, discovery, and 
sharing 

Social network interactions, concept development in group work, 
“on task” contributions in discussions and with artifacts 
Contrary views presented, tone and sentiment of text/audio/video 
reflect questioning and critical orientation (De Liddo et ah, 2011) 
Use of tags, conversation flow across tools/spaces (Rose et ah, 
2007) 

Trails of interactions, networks and search queries (Judd & 
Kennedy, 2011) 

Debate, discourse tools, give and take interactions in knowledge 
development (Pozzi et ah, 2007) 


The media literacies described are still reliant upon an individual student 
demonstrating a certain level of competency in the more traditional literacies (Jenkins 
et ah, 2006). A student who has limited reading and writing proficiency will continue to 
struggle with new media literacies. As depicted in Figure 1, the traditional literacies are 
the foundation for all other literacies. These basics can be evaluated through automated 
content analysis. Textual passages can be extracted from student activity with blogs, 
wikis, traditional writing assignments, discussion forum activity, even Twitter posts. 
However, while this data can be mined and analysed, an understanding of the learning 
design is essential for establishing meaningful indicators and assessment of an 
individual’s proficiency within one or more literacy (Lockyer, Heathcote, & Dawson, 
2013). 

Learning analytics and educational data mining frameworks have been created by 
researchers to describe the range of activity, in terms of techniques and applications, 
undertaken by researchers working with learning-related data (Baker & Yacef, 2009; 
Bienkowski, Feng, & Means, 2012; Siemens, 2013). These suggested frameworks reflect 
a sequential maturing and refinement of analytics work whereby the more recent work 
undertaken by Siemens incorporates and builds upon the earlier studies. Siemens’ 
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framework comprises two related components: analytics techniques and applications. 
Analytic techniques include: modeling; relationship mining; and knowledge domain 
mapping. Analytic applications involve: applications; prediction; personalization and 
adaptive learning; and structured mapping. Mapping the proposed four multiliteracies 
(see Figure 1), experimentation, products/creation, network agility and citizenship, and 
task effectiveness and efficiency, to an existing framework of learning analytics 
techniques and applications (Siemens, 2013) can provide the analytics opportunities as 
described in Table 3. 

Table 3 

Mapping Multiliteracies to Learning Analytics Techniques and Applications 


Multiliteracy 

Learning analytics techniques & applications 

Experimentation 

Modeling 


Knowledge domain mapping 

Products & Creation 

Personalization 


Structured mapping 


Prediction 

Network agility & citizenship 

Relationship mining 


Modeling 

Task effectiveness and efficiency 

Structured mapping 


Prediction 


Conclusion 


With growing interest in data and analytics in the education sector, it is important for 
researchers, educators, learners, and administrators to have tools and techniques that 
go beyond surface-level analytics. The complexity of a social process in learning cannot 
be adequately assessed through basic metrics such as logins, time online, and clicks. 
Multiliteracies draw attention to the skills and attributes learners require to navigate the 
increasingly complex technical, social, cultural, and economic worlds. However, 
traditional models of standardized assessment do little to either promote or effectively 
measure these multiliteracies (I. Brown et al., 2010; Kalantzis et al., 2003). 

This paper builds upon established theoretical models to demonstrate the role learning 
analytics can play in assisting educators and students in developing real-time feedback 
and analytics for evaluating literacies. Thus, the significance of the model resides in its 
capacity to provide deep and nuanced insight into the learning activities of students and 
to merge the boundary between multiliteracies and learning analytics. In this context, 
alternate and diverse assessment techniques and instruments are necessary to better 
align and reflect the technical and information complexity and multimodal learning that 
form the core of 21 st century education. 
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